Data Quality Improvement and Internal Data Audit of the Chinese Neonatal Network Data Collection System

Background: The Chinese Neonatal Network (CHNN) is a nationwide neonatal network that aims to improve clinical neonatal care quality and short- and long-term health outcomes of infants. This study aims to assess the quality of the Chinese Neonatal Network database by conducting an internal audit of data extraction. Methods: A data audit was performed by independently replicating the data collection and entry process in all 58 tertiary neonatal intensive care units (NICU) participating in the CHNN. Eighty-eight data elements selected for re-abstraction were classified into three categories (critical, important, less important), and agreement rates for original and re-abstracted data were predefined. Three to five records were randomly selected at each site for re-abstraction, including one short- (0–7 days), two medium- (8–28 days), and two long-stay (more than 28 days) cases. Agreement rates for each data item were calculated for individual NICUs and across the network, respectively. Results: A total of 283 cases and 24,904 data fields were re-abstracted. The agreement rates for original and re-abstracted data elements were 96.1% overall, and 97.2, 94.3, and 96.6% for critical, important, and less important data elements, respectively. Individual site variation for discrepancies ranged between 0.0 and 18.4% for all collected data elements. Conclusion: The completeness, precision, and quality of data in the CHNN database are high, providing assurance for multipurpose use, including health service evaluation, quality improvement, clinical trials, and other research.


INTRODUCTION
Preterm birth and low birth weight remain the single largest cause of neonatal mortality and morbidity among births worldwide (1). Optimizing neonatal outcomes in preterm newborn infants is a priority internationally (2). Consequently, many regional, national, and international neonatal networks have been established over the past two decades, with the aim of improving clinical neonatal quality of care (3). Standardized neonatal databases that collect neonatal clinical information are critical for transforming the raw data collected into evidencebased conclusions (4) through benchmarking, monitoring and feedback of neonatal outcomes, critical evaluation of clinical care practices and healthcare service delivery, and continuous quality improvement efforts. These databases may also provide robust platforms for neonatal research (5).
The quality of data is essential for good epidemiological study since poor data quality may lead to unreliable conclusions (6). There are several approaches to improve the precision and reliability of data abstracted from medical records including abstractor training, routine communication with abstractors, standardized application of electronic data entry, and data audit (7,8). Data can be manually abstracted from patient medical records into a customized data entry application (8). The use of a clearly written data abstraction manual combined with training of abstractors may decrease the potential for abstractors to apply subjective judgment and impair accuracy of data abstraction (9,10). A major limitation of data abstraction is transcription error due to manual data entry by abstractors (11). Computerized data entry can reduce transcription error since the application is always linked with the electronic database (8). Incorporation of data validation into the computer application can also increase the accuracy of data collection. Routine internal data audit is another efficient method for checking the precision and reliability of data abstraction, especially when data are abstracted by multiple abstractors at different sites, which may result in interor intrarater variability (12). During internal data audit, data are reabstracted by the same or different abstractors to determine the disagreement between the original abstraction and reabstraction which is a quantitative evaluation of data precision (10). Random sampling applied during the data audit can reduce the time, cost, and human labor required (12,13).
The Chinese Neonatal Network (CHNN) was established in 2019, with the aim of creating a trustworthy national source of neonatal clinical information for research, benchmarking, quality improvement, and policy making. Consequently, quality assurance of the database is particularly important to make sure that the data collected are of high quality. The objective of this manuscript is to report the results of an internal audit of the CHNN database using a prospective cross-sectional study approach. Only deidentified data are transferred to the data coordinating center. All data protocols and procedures comply with national, provincial, and local regulations for protecting patients' personal privacy and confidentiality. A custom-built, stand-alone system based on MS Access was developed and used for data collection and transfer. Similar to the Canadian Neonatal Network, data are prospectively collected on maternal and neonatal demographics, antenatal and birth history, NICU admission, NICU treatment, health outcome, and hospital discharge (12). Trained abstractors in each hospital abstract data from patient medical records directly into dedicated computers using a standard abstractor manual of operations and procedures, with standardized definitions of variables.

METHOD
Each participant hospital has at least one dedicated abstractor, who is supervised by hospital site investigators and a single coordinator at the CHNN Coordinating Center, who answers questions about the process of data entry. Automated validation checks are incorporated in the data-entry software, and patient identifiers are stripped before data are submitted to the coordinating center. The central coordinator also checks the database regularly and writes the report for data quality. After submission, data are automatically checked by an error checking program and potential errors are fed back to each site for data recheck and correction (Figure 1). A randomized data audit was conducted after the data correction process to assess the quality of the data.
Data collection in the CHNN database started from Jan. 1st, 2019. The audited data was obtained from newborns who were admitted to CHNN NICUs between Jan 1, 2019 and Nov 31, 2019. During this time, there were 8,103 validated cases in the CHNN database.

Sampling
Three to five cases were randomly selected in each hospital based on the number of NICU admissions, i.e., three cases for smaller hospitals and five cases for larger hospitals. Within each hospital, stratified random sampling was based on duration of hospitalization in three categories-short (0-7 days), medium (8-28 days), and long (>28 days) stay. The number of cases selected for audit in each category was prorated according to the distribution of cases in each category among the 58 CHNN sites. Accordingly, 53 short-stay, 108 medium-stay, and 122 long-stay cases were selected, for a total sample size of 283 cases for the data audit.

Variable Selection for Data Reabstraction
We used the method of Shah et al. (12) to divide all data elements, including demographic, procedural, diagnostic, and outcome information, into three categories (critical, important, less important) based on the ideal level of required reliability of each item. The expected agreement rate of the three categories were ≥95, ≥90, and ≥85% concordance for critical, important, and less-important elements, respectively, which is the tolerable criteria for the data abstraction. Thirty elements were categorized as critical, 28 elements as important, and 30 elements as less important. The criteria for inclusion of these variables in each category were based on the need to test the precision of target variable and the possibilities of occurrence of subjective judgment. If the data elements did not meet the expected agreement rate, the coordinating center will analyze the potential problems in the process of data collection by communication with the site abstractor and investigators, and check the definition, structure, and purpose of variable itself. Furthermore, we also checked to ensure that selected variables were representative of all 360 variables in the entire CHNN databases by calculating the total agreement rate for 360 variables.

Distribution of Sampling Results
After selection of variables, the coordinating center used the sampling criteria to randomly select cases from the original database for reabstraction and informed the sites accordingly. Within the following month, abstractors used a separate data reabstraction software to reenter data for cases randomly selected for data reabstraction. Abstractors were not permitted to revisit and review the original abstraction for checking of accuracy of data entry. The average time required for reabstraction of the 88 variables required for the audit was 50 min. This contrasts with the average of 60 min required for the 360 variables that are routinely abstracted for a typical patient and indicates the complexity of the variables chosen for reabstraction. Consequently, the total extra workload averaged between 2 and 3 h for each participating CHNN hospital. All CHNN hospitals participated in the data audit.

Data Analysis
Data analysis and random sampling were conducted using SAS version 9.4. Before data analysis, records of selected cases were extracted from the original CHNN database and categorized into the three abovementioned categories. The total agreement rates of each variable in these three categories were analyzed and evaluated using the following formula: Agreement rate = Number of agreement between all original and reabstracted case total number of selected cases For each participant site, the range of agreement rate was reported, which reflects the percentage of reabstracted record that agreed with original abstraction in each hospital of CHNN. Descriptive analysis was applied to report the median gestational age and length of stay of cases.

Pilot Study
A pilot study was conducted at the Children's Hospital of Fudan University 3 months prior to the CHNN audit exercise to test the data reabstraction software and ensure that the audit exercise could be carried out smoothly. We sampled three cases at random for the pilot study. Data analysis of the pilot study confirmed that the process could be executed efficiently.

Ethics
Ethics approval was obtained from the Children's Hospital of Fudan University to collect, store, and analyze the data from newborn infants admitted to all participant sites of CHNN. Ethics approvals were also obtained from all participating CHNN sites for local data collection, storage, and transfer of patient data to the CHNN Coordinating Center for analysis. All necessary health information privacy and security procedures were followed and conformed with local, provincial, and national requirements and guidelines. Prior to data transfer to the Coordinating Center, all patient identifiers were stripped to ensure privacy.  Table 2). Data elements that exceeded the pre-established <5% discrepancy threshold were receipt of antenatal corticosteroids, patent ductus arteriosus diagnosed or treated, and retinopathy of prematurity. The disagreement rate for antenatal corticosteroids was twice as high at Children's Hospitals (10 cases) compared with Maternity Hospitals (five cases) due to poor information transfer from the birth hospitals. There were 11 cases where misclassification for the question "Was screening for ROP performed or not?" led to errors in ROP-related variables.

RESULTS
For important data elements, the individual site agreement rate ranged from 78.6 to 100% ( Table 1). For individual important data elements, the agreement rate ranged from 100% for laparotomy, ventriculoperitoneal shunt, and exchange transfusion to 85.9% (range 0-100%) for suspected chorioamnionitis ( Table 3). Data elements which exceeded the threshold discordance rate of 10% were course and timing of   antenatal corticosteroids, receipt of more than one course of antenatal corticosteroids, and suspected chorioamnionitis. The discrepancy was higher among outborn compared with inborn cases, for receipt of more than one course of AC (17 outborn vs. 12 inborn), timing of AC, and suspected chorioamnionitis (23 outborn vs. 17 inborn). For less-important data elements, the individual site agreement rate ranged from 82.7 to 100.0% ( Table 1). For individual less-important data elements, the agreement rate ranged from 100% for procedures like ostomy, necrotizing enterocolitis peritoneal drainage, gastrostomy, and tracheostomy to 87.6% (range, 20.0-100.0%) for stimulus response appropriate for GA ( Table 4). None of the data elements in this category crossed the threshold discordance rate of 15%.
The overall agreement of all 360 data variables in the entire database was 96.3% for the n = 98,100 variables. The agreement rate for cases with short duration of hospitalization was 96.0% (n = 20,144 variables), medium duration of hospitalization was 95.9% (n = 42,549 variables), and long duration of hospitalization was 96.1% (n = 44,890 variables).

DISCUSSION
To our knowledge, this is the first study of data quality in a neonatal database in China. This internal audit with participation of all CHNN sites showed a high rate of agreement between the reabstracted and original data collected for the CHNN database, which is the largest neonatal database of its kind in China, with participation of 58 NICUs in 25 provinces throughout China. The overall agreement rate was high at 96.1%, and the disagreement rates of 2.8% for critical variables, 5.7% for important variables, and 3.3% for less-important variables, met their prescribed targets of <5, <10, and <15%, respectively. There was low site-to-site variation in the disagreement rate, ranging from  perfect agreement to 18.4% disagreement for all collected data elements. These results demonstrate that the CHNN data have high precision and reliability and is suitable for multiple uses, including research. Our results are comparable with audits performed by large neonatal networks in other countries. The Canadian Neonatal Network reported a high precision of data quality collection (agreement rate of 96.9% overall, 98% for critical, 96.1% for important, and 96.3% for less-important data elements) with small individual site variation for discrepancies (0.2-12.8%) through a similar internal audit process (12). The Vermont-Oxford Trials Network conducted a similar evaluation from 635  The disagreement rate ranged from 1.3% for birth date to 8.8% for discharge date, and 90% of disagreements were due to errors in transcription or interpretation, rather than data keying errors (14). An assessment of the neonatal database maintained by the United Kingdom National Neonatal Research Database revealed low discordance rate of <5% for patient characteristics, treatment, and clinical outcomes (5). We found that disagreement between the reabstracted and original data were mainly due to missing maternal data and incomplete records in the patient medical charts rather than poor abstraction of data. Missing data secondary to poor patient medical chart documentation by healthcare providers is a documented phenomenon (15). In our study, many outborn infants in Children's Hospitals in CHNN lack maternal information such as the use of AC (including timing and course) and chorioamnionitis, because maternal information is often not routinely provided when the infant is transferred from the maternity hospital. Although statistical methods such as multiple imputations can be applied when maternal data is missing, these methods have their own limitations. Another common reason for disagreement between the reabstracted and original data was the ambiguity of some information recorded in the electronic medical record (EMR). For example, in some CHNN sites, abstractors had difficulty distinguishing suspected from histologically diagnosed chorioamnionitis because only chorioamnionitis was recorded in the EMR. Misinterpretation of the definition of chorioamnionitis by some abstractors added to the confusion and further compounded the error. Although this was the first year of data collection in CHNN, we did not find significant data keying errors. Horbar et al. reported that data keying errors in the Vermont-Oxford Trials Network decreased from 13.7 to 3.7% over time with improvements in data keying procedures (14). This is encouraging, and we anticipate that CHNN data quality will further improve as our procedures are streamlined over time.
The high precision and reliability of the database is due at least in part to the multiple layers of quality assurance built into the data collection system (16). These include central coordination, dedicated data entry personnel with uniform training by the coordinating center, a single manual of operations and protocols with standardized definitions, direct data entry into a computer, use of a unique data entry program with built-in error checking, multiple levels of error checking after data entry, feedback of errors to participating sites with a protocol for correcting errors, and annual audit of the data collected. In this regard, this audit was also useful because it highlighted data problems that were not previously apparent and CHNN has since initiated a networkwide effort to resolve the difficulties identified, such as missing maternal information and ambiguity of medical chart records. Consequently, we anticipate that the precision of the database will further improve in the future.
There are several limitations to this internal audit. Our audit exercise involved only a small random sample and was only conducted once per year. However, to reabstract and audit all cases in the database would be prohibitively time consuming, expensive, and impractical. Consequently, most databases only conduct small random sampling audits similar to our study (12,17). An external audit using different data abstractors is considered more reliable than an internal audit. However, it is labor intensive, costly, and time consuming because external abstractors need to deal with unfamiliar medical record systems and protocols, and they collect data retrospectively rather than prospectively. Consequently, external audits are seldom employed in audits of network databases (12).
In conclusion, our audit demonstrated that data from the CHNN data collection system show high precision and reliability. With the implementation of measures designed to improve data quality based on deficiencies identified in this audit, we expect that data quality will further improve. As our network matures, periodic data audit will be essential to ensure the reliability of our database for research, and funding, manpower, and resources will be needed to support the practice of data quality improvement.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding authors.

ETHICS STATEMENT
The Ethics Committee from the Children's Hospital of Fudan University has approved CHNN to collect, store and analyse the overall data from all participant sites of CHNN. For individual sites, there is also approval from the ethics committee for local data collection and storage. Health information for each case will be well protected and maintained at local sites with a unique ID, which will not be submitted in the coordinate center. Since audit practice is the part of the current exercise of the project, specific approval will not be warranted.

AUTHOR CONTRIBUTIONS
JS, SL, and LD: study concept and design. JS, XC, SL, and LD: acquisition, analysis, or interpretation of data. JS and XC: drafting of the manuscript. XC, YW, and XG: statistical analysis. TY and YL: administrative, technical, or material support. JS, WenZ, CC, SL, and LD: study supervision. All authors are critical revision of the manuscript for important intellectual content. All authors contributed to the article and approved the submitted version.